1. The Field of the Invention
The present invention relates to testing technology. More specifically, the present invention relates to isolating the evaluation of actual test results against expected test results from the test module that generates the actual test results.
2. Background and Related Art
Computers have revolutionized the way we work and play. There are an enormous variety of functions and applications that may be implemented by a general purpose computing system in response to the execution of a software application. The utility and functionality of the computing system does, however, rely on the proper coding of the source code that was compiled or interpreted into the binary instructions that are actually executed by the processor. If there is a coding error, this will often result in a deviation from expected functionality.
Extraordinary efforts are currently in place to reduce the number of unexpected performance deviations in many software programs before and after the software application are shipped to market. However, the creativity of software programmers and designers has led to increasingly complex and powerful software applications. As the complexity of the software application increases, so often does the number of lines of source code needed to generate the software application. As a result, a software application is often tested before release to the general public to help ensure that the software application operates as intended.
In some testing environments, a test developer develops tests that test the performance of one or more functions of a software application. For example, objects (methods, properties, classes, etc.) in a software application are often tested to attempt to identify combinations of input values that will cause output failures. One conventional method of testing software is to have a human tester (which may or may not be the software application programmer) manually analyze an object (or specification for the object) to identify input fields for the object. The tester then manual enters input field values to cause the object to generate an output (e.g., one or more output field values). The tester then determines if the test passes or fails based on the generated output.
However, manually analyzing a method (or specification) and manually inputting field values is time consuming and can introduce human error into the testing process. For example, if a tester enters an inappropriate input field value, the validity of generated output can be affected. Further, the time consumed to analyze and input field values for a number of interrelated objects, such as, for example, those of a software module or application, can be quite significant as each object must be individually analyzed and input field values manually generated for each method.
Accordingly, in part due to the inefficiencies of manual testing, automated testing programs have been developed. An automated test program essentially automates the entering of input field values, gathering of resulting output, and determining if a test passed or failed. Typically, a test developer with knowledge of an object to be tested develops an automated testing program to test the object. A test developer can use their knowledge of the object to test the object with input field values that are likely to expose deficiencies in or unintended operation of the object. Based on knowledge of the object the test developer can also identify the expected resulting output of the object. Thus, the test developer can configure the testing program to compare actual resulting output to expected resulting output and, based on the comparison, determine if an object is operating as intended.
A tester (which may or may not be the test developer) can then execute the test program to test the object. Executing the test program causes the test program to provide an indication of whether the test passed or failed (and thus whether the object is operating as intended). Unfortunately, automated test programs are typically configured to operate in a single environment. For example, a test developer may develop a test program for testing a word processor object in a first environment that includes a Windows operating system, a Pentium 3 processor, and has 128 MB system memory. Accordingly, when the test program is executed in this first environment, an indication of whether the word processor object passed or failed may be valid.
However, due to inevitable differences in both hardware and software components of a computer system, a test program designed to test the word processor object in one environment may not provide valid results in one or more other environments. Further, a test program may have no way to identify the test program's operating environment. For example, executing the test program in a second environment that includes a Unix operating system (or even a different version of the Windows operating system), a Pentium 4 processor, and/or 56 MB of memory, may provide different resulting output than the first environment. This different resulting output may be resulting output indicating that the word processor object is operating as intended in the second environment. However, since the different resulting output is not resulting output that indicates the word processor is operating as intended in the first environment, the test program may inappropriately indicate the word processor object is not operating as intended in the second environment.
To configure the test program to provide a valid pass/fail indication for the second environment, the test developer may be required to include new instructions in the test program. Further, as new environments are developed as a result of technological advances, the developer may also be required to develop additional instructions for these new environments. Thus, a test developer may be required to essentially continually maintain a test program when an object tested by the test program is used in new environments. Accordingly, what would be advantageous are mechanisms for evaluating test results that do not depend on the test module under execution generating the actual test results.